<h3 id="table-of-contents">Table of Contents</h3>
<ul>
<li><a href="#commands">Commands</a>
<ul>
<li><a href="#modify-pool-replica-count">Modify Pool Replica Count</a></li>
<li><a href="#show-current-space-used">Show current space used</a></li>
<li><a href="#show-status">Show status</a></li>
<li><a href="#list-create-and-delete-volumes">List, create and delete volumes</a></li>
<li><a href="#presenting-a-volume-to-this-host">Presenting a volume to this host</a></li>
<li><a href="#snapshots">Snapshots</a></li>
<li><a href="#save-rbd-as-file">Save RBD as file</a></li>
<li><a href="#python-automation">Python Automation</a></li>
</ul>
</li>
</ul>
<h1 id="commands">Commands</h1>
<h2 id="modify-pool-replica-count">Modify Pool Replica Count</h2>
<p>Show and modify the replica count (size) in a given pool.</p>
<p>You can do the same with <code>min_size</code> if needed.</p>
<pre><code class="language-bash"># Show the current replica count
#   ceph osd pool get &#x3C;pool name> size
ceph osd pool get images size

# Set the replica count for a given pool
#   ceph osd pool set &#x3C;pool name> size &#x3C;number of replicas>
ceph osd pool set images size 2
</code></pre>
<h2 id="show-current-space-used">Show current space used</h2>
<pre><code class="language-bash">ceph df
</code></pre>
<h2 id="show-status">Show status</h2>
<pre><code class="language-bash"># Show the status right now
ceph -s

# Show the status and watch it
ceph -w
</code></pre>
<h2 id="list-create-and-delete-volumes">List, create and delete volumes</h2>
<pre><code class="language-bash"># LIST
# rbd ls -p &#x3C;pool name>
rbd ls -p volumes

# CREATE
#   rbd create --size &#x3C;size in gb> &#x3C;pool>/&#x3C;volume id>
rbd create --size 10 volumes/myVolume

# DELETE
#   rbd rm &#x3C;pool>/&#x3C;volume id>
rbd rm volumes/myVolume
</code></pre>
<h2 id="presenting-a-volume-to-this-host">Presenting a volume to this host</h2>
<pre><code class="language-bash"># PRESENT TO HOST
#   rbd map &#x3C;pool>/&#x3C;volume id>
rbd map volumes/myVolume

# LIST MAPPED
rbd showmapped

# REMOVE FROM HOST
#   rbd unmap &#x3C;pool>/&#x3C;volume id>
rbd unmap volumes/myVolume
</code></pre>
<p>Once you've presented the volume you can use it like this:</p>
<pre><code class="language-bash">mkfs.ext4 /dev/rbd0
mkdir /data
mount /dev/rbd0 /data
</code></pre>
<h2 id="snapshots">Snapshots</h2>
<pre><code class="language-bash"># LIST SNAPSHOTS
#   rbd snap ls &#x3C;pool>/&#x3C;volume id>
rbd snap ls volumes/myVolume

# REVERT TO SNAPSHOT
# rbd snap rollback &#x3C;pool>/&#x3C;volume id>@&#x3C;snapshot id>
rbd snap rollback volumes/myVolume@sampleSnapshot

# DELETE SNAPSHOT
#   rbd snap rm &#x3C;pool>/&#x3C;volume id>@&#x3C;snap id>
rbd snap rm volumes/myVolume@sampleSnapshot
</code></pre>
<h2 id="save-rbd-as-file">Save RBD as file</h2>
<pre><code class="language-bash">#   rbd export &#x3C;pool>/&#x3C;volume> &#x3C;filename>
rbd export volumes/myVolume myVolume.raw
</code></pre>
<h2 id="python-automation">Python Automation</h2>
<p>To automate any of the above from Python, the <code>rados</code> and <code>rbd</code> libraries
wrap the same operations. See <a href="/python-ceph.html">Python Ceph Examples</a> for
pool management, volume creation, and snapshot handling via the Python API.</p>


<p><strong>This is a Reference Page:</strong> Reference pages are collections of short bits
of information that are useful to look up but not long or interesting enough to
justify having their own dedicated post.</p>
<p>This reference page lists some of the errors that I've personally encountered,
and how I fixed them. I might also include errors I never fixed in case anyone
wants to talk about it.</p>
<hr>
<h3 id="table-of-contents">Table of Contents</h3>
<ul>
<li><a href="#docker-login-fails">Docker Login Fails</a></li>
<li><a href="#runaway-memory-usage-in-ceph-osd-process">Runaway memory usage in Ceph OSD process</a></li>
<li><a href="#chrome-blocks-https127001-with-neterr_cert_revoked">Chrome blocks https://127.0.0.1 with NET::ERR_CERT_REVOKED</a></li>
<li><a href="#failed-command-aws-ecr-get-login">Failed command: aws ecr get-login</a></li>
<li><a href="#pure-storage-iscsi-with-kolla-openstack---fdisk-freezes">Pure Storage iSCSI with Kolla OpenStack - fdisk freezes</a></li>
<li><a href="#weird-chars-when-pasting-on-mac-os">Weird chars when pasting on Mac OS</a></li>
</ul>
<h1 id="docker-login-fails">Docker Login Fails</h1>
<pre><code>Error saving credentials: error storing credentials - err: exit status 1, out: `Cannot autolaunch D-Bus without X11 $DISPLAY`
</code></pre>
<p>I had this happen after running both Ceph-Ansible then Kolla-Ansible on an Ubuntu server, so it could be either of those that set up the conditions for this to happen. <a href="https://github.com/docker/cli/issues/1136">this solution</a> worked for me:</p>
<pre><code class="language-bash">apt-get install -y gnupg2 pass
</code></pre>
<hr>
<h1 id="runaway-memory-usage-in-ceph-osd-process">Runaway memory usage in Ceph OSD process</h1>
<p>It looked like a memory leak. Each OSD process was using around 30GB of RAM.</p>
<p>It turns out that Ceph-Ansible has the following logic when generating its
<code>ceph.conf</code> file:</p>
<pre><code># this math
roles/ceph-config/templates/ceph.conf.j2:{% set _osd_memory_target = (ansible_memtotal_mb * 1048576 * hci_safety_factor / _num_osds) | int %}
# sets osd memory target
roles/ceph-config/templates/ceph.conf.j2:osd memory target = {{ _osd_memory_target | default(osd_memory_target) }}
</code></pre>
<p>In my case, this set the value WAY too high. In one of my deployments, it was
set to <code>37853425827</code>, which is like 35GB (PER OSD SERVICE!).</p>
<p>The 4GB default is much more sane, <code>4294967296</code>.</p>
<pre><code># ceph.conf
[osd]
osd memory target = 4294967296
</code></pre>
<hr>
<h1 id="chrome-blocks-https127001-with-neterr_cert_revoked">Chrome blocks <a href="https://127.0.0.1">https://127.0.0.1</a> with NET::ERR_CERT_REVOKED</h1>
<p>This comes up when I reverse tunnel to a remove web server, forwarding it to my workstation. Chrome has decided to not let you do that. They give this errror:</p>
<pre><code class="language-text">Your connection is not private
Attackers might be trying to steal your information from 127.0.0.1 (for example, passwords, messages, or credit cards). Learn more
NET::ERR_CERT_REVOKED
</code></pre>
<p>To bypass that, open <code>chrome://flags/#allow-insecure-localhost</code> and set allow-insecure-localhost to enabled. Click relaunch. This will nuke your current browser…</p>
<p>Once it relaunches you'll be able to open the site.</p>
<hr>
<h1 id="failed-command-aws-ecr-get-login">Failed command: aws ecr get-login</h1>
<p>This error was caused by a bad NTP config. Renew the NTP lease to fix it.</p>
<pre><code>An error occurred (InvalidSignatureException) when calling the GetAuthorizationToken operation: Signature expired: 20190911T174538Z is now earlier than 20190912T004835Z (20190912T010335Z - 15 min.)
</code></pre>
<hr>
<h1 id="pure-storage-iscsi-with-kolla-openstack---fdisk-freezes">Pure Storage iSCSI with Kolla OpenStack - fdisk freezes</h1>
<p>This was a weird one. When using the Kolla containers project to make a
Cinder-volume container that has the <code>purecinder</code> plugin, I was able to get
OpenStack to create the volumes in Pure but it couldn't mount them to the hosts
for things like creating a volume with the <code>--image</code> argument. That made the
volumes I could create basically useless.</p>
<p>These errors were in the logs:</p>
<pre><code class="language-text"># Lots of these in cinder's log
Trying to connect to iSCSI portal

# also from cinder's log, also recurring:
iscsiadm stderr output when getting sessions: iscsiadm: No active sessions

# in DMESG
FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
</code></pre>
<p>The weirdest part was that the <code>fdisk -l</code> command would totally lock up and
the process running it couldn't be killed even with <code>kill -9</code>. The iSCSI volume
never got properly mounted but Pure support said they could see some data
coming into it.</p>
<p><strong>Root cause</strong>: MTU mismatch. Of course. I had set MTU 9000 on the iSCSI ports,
my host ports, and the switchports heading to the Pure Storage iSCSI
interfaces. I had forgotten to set it on the switch heading to the Dell servers
being used for OpenStack.</p>
<hr>
<h1 id="weird-chars-when-pasting-on-mac-os">Weird chars when pasting on Mac OS</h1>
<p>When you paste sometimes you get <code>^[[200~</code> at the start and <code>^[[201~</code> at the
end of the pasted content. Or maybe <code>0~</code> and <code>1~</code>.</p>
<p>To resolve, run:</p>
<pre><code class="language-bash">printf "\e[?2004l"^C
</code></pre>


<p><strong>This is a Reference Page:</strong> Reference pages are collections of short bits
of information that are useful to look up but not long or interesting enough to
justify having their own dedicated post.</p>
<p>This reference page contains operations-related mini-guides and minor posts.</p>
<hr>
<h3 id="table-of-contents">Table of Contents</h3>
<ul>
<li><a href="#find-which-active-passive-bond-member-is-active">Find which active-passive bond member is active</a></li>
<li><a href="#convert-p7b-certificate-to-pem-format">Convert p7b certificate to pem format</a></li>
<li><a href="#journalctl-commands">Journalctl commands</a></li>
<li><a href="#get-real-ram-use-of-processes">Get real RAM use of processes</a>
<ul>
<li><a href="#of-ceph">of ceph</a></li>
<li><a href="#of-vms">of VMs</a></li>
</ul>
</li>
<li><a href="#bypass-chromes-self-signed-cert-warning">Bypass Chrome's self-signed cert warning</a></li>
<li><a href="#create-a-virtual-loopback-volume-on-ubuntu">Create a virtual loopback volume on Ubuntu</a>
<ul>
<li><a href="#use-systemd-to-re-map-loopback-on-reboot">Use Systemd to re-map loopback on reboot</a></li>
</ul>
</li>
<li><a href="#test-mtu-from-ubuntu">Test MTU from Ubuntu</a></li>
<li><a href="#find-big-files-on-mac-os">Find big files on Mac OS</a></li>
<li><a href="#renew-ntp-lease-ubuntu">Renew NTP Lease Ubuntu</a></li>
<li><a href="#create-aws-ecr-registry-if-not-found">Create AWS ECR Registry If Not Found</a></li>
<li><a href="#passwordless-sudo-on-ubuntu-server">Passwordless sudo on Ubuntu Server</a></li>
<li><a href="#vim-tasks">VIM Tasks</a>
<ul>
<li><a href="#find--replace">Find &#x26; Replace</a></li>
<li><a href="#spell-check-commands">Spell-Check commands</a></li>
</ul>
</li>
<li><a href="#use-an-insecure-docker-registry">Use an Insecure Docker Registry</a></li>
<li><a href="#ansible-ignore-errors">Ansible ignore errors</a></li>
<li><a href="#how-to-tail--f-on-dmesg">How to tail -f on dmesg</a></li>
<li><a href="#ubuntu-downgrade-a-package">Ubuntu Downgrade a package</a></li>
<li><a href="#make-ansible-run-python-3-on-target-nodes">Make Ansible run Python 3 on target nodes</a></li>
<li><a href="#bash-print-all-but-first-line">Bash: Print all but first line</a></li>
<li><a href="#bash-run-code-whenever-a-script-exits">Bash: Run code whenever a script exits</a></li>
</ul>
<h1 id="find-which-active-passive-bond-member-is-active">Find which active-passive bond member is active</h1>
<pre><code>cat /proc/net/bonding/bond0
</code></pre>
<hr>
<h1 id="convert-p7b-certificate-to-pem-format">Convert p7b certificate to pem format</h1>
<pre><code class="language-bash">openssl pkcs7 -print_certs -in certnew.p7b -out cert.pem
</code></pre>
<p>To generate a CSR with Subject Alternative Names (required to fix
<code>ERR_CERT_COMMON_NAME_INVALID</code>), see
<a href="/openssl-csr-san.html">Creating a CSR with a SAN</a>.</p>
<hr>
<h1 id="journalctl-commands">Journalctl commands</h1>
<p>Watch the logs as they come in:</p>
<pre><code class="language-bash">journalctl -f
</code></pre>
<p>When journalctl logs too much and you need to pair it down:</p>
<pre><code class="language-bash"># to 30 days
journalctl --vacuum-time=30d

# to 500MB
journalctl --vacuum-size=500M
</code></pre>
<p>Show for a specific service</p>
<pre><code class="language-bash">journalctl ceph*
</code></pre>
<hr>
<h1 id="get-real-ram-use-of-processes">Get real RAM use of processes</h1>
<p>You can always use top or htop or whatever. I use this to specifically get the
info about those processes.</p>
<h2 id="of-ceph">of ceph</h2>
<pre><code>ps aux | grep ceph | grep -v -e grep -e qemu -e "\[" | while read line; do bin=$(echo $line | awk '{print $11}'); pid=$(echo $line | awk '{print $2}'); id=$(echo $line | awk '{print $16}'); size=$(grep 'VmSize' /proc/$pid/status); echo -e "pid: $pid, \tbin: $bin, \tid: $id,    \tRAM: ($size)"; done
</code></pre>
<h2 id="of-vms">of VMs</h2>
<p>for KVM</p>
<pre><code>ps aux | grep qemu | grep -v -e grep| while read line; do bin=$(echo $line | awk '{print $11}'); pid=$(echo $line | awk '{print $2}'); guest=$(echo $line | awk '{print $13}'); size=$(grep 'VmSize' /proc/$pid/status); echo -e "pid: $pid, \tbin: $bin, \tguest: $guest,   \t\tRAM: ($size)" ; done
</code></pre>
<h1 id="bypass-chromes-self-signed-cert-warning">Bypass Chrome's self-signed cert warning</h1>
<p>As of Mac Catalina, developing a site using self-signed certs is a bigger pain
than before. Yeah you can go get the key and add it, but that's a nuisance.
It looks like the Chrome team built something they call an
"interstitial bypass keyword" to deal with that. That's one cool feature name.</p>
<p><strong>Obvious disclaimer</strong> If you're not a developer doing this to access your own
site, don't do this.</p>
<p>To bypass the warning, type</p>
<pre><code>thisisunsafe
</code></pre>
<p>Apparently it used to be <code>badidea</code> but they rotated it since it got too well
known. They'll probably do it again someday. I suspect people are using this
on sites they "trust" without really knowing any better and getting hurt by
it.</p>
<hr>
<h1 id="create-a-virtual-loopback-volume-on-ubuntu">Create a virtual loopback volume on Ubuntu</h1>
<p>Useful for simulating a real drive when building a test environment. I use this to test OpenStack in a VM as a Ceph or LVM back-end.</p>
<pre><code class="language-bash"># create empty file of all zeros
sudo dd if=/dev/zero of=/root/virtual-disk bs=1M count=512
# show list of loopback devices. Note its not there
losetup --list
# create the loopback device
losetup /dev/loop0 /root/virtual-disk
# confirm it worked
losetup --list
</code></pre>
<p>You can now treat it like a normal volume. Mount it, put a filesystem on it, whatever. Example:</p>
<pre><code class="language-bash">pvcreate /dev/loop0
vgcreate cinder-volumes /dev/loop0
</code></pre>
<h2 id="use-systemd-to-re-map-loopback-on-reboot">Use Systemd to re-map loopback on reboot</h2>
<p>I found <a href="https://unix.stackexchange.com/questions/418322/persistent-lvm-device-with-loopback-devices-by-fstab">this useful systemd unit example</a> that didn't quite work for me, but was close. Here's mine:</p>
<p><code>vi /etc/systemd/system/loops-setup.service</code></p>
<pre><code class="language-text">[Unit]
Description=Setup loopback devices

DefaultDependencies=no
Conflicts=umount.target

Requires=lvm2-lvmetad.service mnt-host.mount
Before=local-fs.target umount.target
After=lvm2-lvmetad.service mnt-host.mount

[Service]
ExecStart=/sbin/losetup /dev/loop0 /root/virtual-disk
ExecStop=/sbin/losetup -d /dev/loop0

RemainAfterExit=yes
Type=oneshot

[Install]
WantedBy=local-fs-pre.target
</code></pre>
<p>Then enable it</p>
<pre><code class="language-bash">systemctl enable loops-setup
</code></pre>
<p>Reboot, then check <code>vgdisplay</code> (if that's your use case). It'll be there.</p>
<hr>
<h1 id="test-mtu-from-ubuntu">Test MTU from Ubuntu</h1>
<p>You would not believe the strange problems that an MTU issue can be the root cause of.</p>
<pre><code class="language-bash">ping -M do -s 1472 &#x3C;remote server>
ping -M do -s 8900 &#x3C;remote server>
</code></pre>
<hr>
<h1 id="find-big-files-on-mac-os">Find big files on Mac OS</h1>
<p>My SSD is always almost full. Here are useful commands for cleaning up:</p>
<pre><code class="language-bash"># Find all the files over 1G
sudo find / -type f -size +1G -exec ls -lh {} \; | awk '{ print $9 ": " $5 }' 2>/dev/null

# List files in Downloads directory by size
ls -lhS ~/Downloads
</code></pre>
<h1 id="renew-ntp-lease-ubuntu">Renew NTP Lease Ubuntu</h1>
<pre><code class="language-bash">service ntp stop
ntpd -gq
service ntp start
</code></pre>
<hr>
<h1 id="create-aws-ecr-registry-if-not-found">Create AWS ECR Registry If Not Found</h1>
<pre><code class="language-bash">aws ecr describe-repositories --repository-names $repo_name \
    || aws ecr create-repository --repository-name $repo_name
</code></pre>
<hr>
<h1 id="passwordless-sudo-on-ubuntu-server">Passwordless sudo on Ubuntu Server</h1>
<p>Run <code>visudo</code> as root.</p>
<pre><code class="language-bash"># as root
export EDITOR=vim
visudo
</code></pre>
<p>Add the user (replace <code>exampleUser</code>) at the end.</p>
<pre><code class="language-text">exampleUser ALL=(ALL) NOPASSWD: ALL
</code></pre>
<p>For those who don't use vim, a quick reminder:
<code>i</code> enter to insert mode , <code>[esc]</code> to exit insert mode, <code>:wq</code> to save and quit.</p>
<hr>
<h1 id="vim-tasks">VIM Tasks</h1>
<h2 id="find--replace">Find &#x26; Replace</h2>
<pre><code class="language-bash"># replace findThis with ReplaceWithThis
:%s/findThis/ReplaceWithThis/g
</code></pre>
<h2 id="spell-check-commands">Spell-Check commands</h2>
<pre><code class="language-text"># enable/disable
:set spell spelllang=en_ca
:setlocal spell spelllang=en_ca
:set nospell

# next/last
]s
[s

# suggest change
z=

# add/remove word from dictionary
zg
zw`
</code></pre>
<hr>
<h1 id="use-an-insecure-docker-registry">Use an Insecure Docker Registry</h1>
<p>To use a registry with no HTTP cert:</p>
<p><code>vi /etc/docker/daemon.json</code></p>
<pre><code class="language-json">{
    "insecure-registries" : ["myregistrydomain.com:5000"]
}
</code></pre>
<hr>
<h1 id="ansible-ignore-errors">Ansible ignore errors</h1>
<p>The command is <code>ignore_errors: yes</code></p>
<p>Just throw this into the task that's failing. It will still fail, but the play
won't stop.</p>
<p>Example:</p>
<pre><code class="language-yml">- name: Ensure the docker service is running
  script: ../tools/validate-docker-execute.sh
    ignore_errors: yes
</code></pre>
<hr>
<h1 id="how-to-tail--f-on-dmesg">How to tail -f on dmesg</h1>
<pre><code class="language-bash">dmesg -wH
</code></pre>
<hr>
<h1 id="ubuntu-downgrade-a-package">Ubuntu Downgrade a package</h1>
<pre><code class="language-bash"># Find the available versions under Versions:
apt-cache showpkg &#x3C;package>

# Install one of those
apt-get install &#x3C;package>=&#x3C;version>
</code></pre>
<hr>
<h1 id="make-ansible-run-python-3-on-target-nodes">Make Ansible run Python 3 on target nodes</h1>
<p>Useful for OS that ship with python3 but not python2.
Edit the inventory, find a group of the groups and apply this var.</p>
<pre><code class="language-ini">[baremetal:vars]
ansible_python_interpreter=/usr/bin/python3
</code></pre>
<hr>
<h1 id="bash-print-all-but-first-line">Bash: Print all but first line</h1>
<p>Use <code>tail -n +2</code>.
This was unintuitive to me because +1 seems like it should do it.</p>
<pre><code class="language-bash"># Example
docker images | awk '{print $3}' | tail -n +2
</code></pre>
<hr>
<h1 id="bash-run-code-whenever-a-script-exits">Bash: Run code whenever a script exits</h1>
<p>Use trap.</p>
<ul>
<li><a href="http://redsymbol.net/articles/bash-exit-traps/">This is a nice post covering lots of examples</a>.</li>
<li><a href="https://github.com/GoogleCloudPlatform/endpoints-quickstart/blob/master/scripts/deploy_api.sh">Here's where I first saw it used</a></li>
</ul>
<pre><code class="language-bash">temp_file=$(mktemp)

cleanup() {
  rm "$temp_file"
}

trap cleanup EXIT
</code></pre>


<p><strong>This is a Reference Page:</strong> Reference pages are collections of short bits
of information that are useful to look up but not long or interesting enough to
justify having their own dedicated post.</p>
<p>This page covers how to do some things in Git.</p>
<hr>
<h3 id="table-of-contents">Table of Contents</h3>
<ul>
<li><a href="#use-vim-for-commits">Use Vim for commits</a></li>
<li><a href="#rename-branch">Rename Branch</a></li>
<li><a href="#roll-back-merge">Roll Back Merge</a>
<ul>
<li><a href="#option-1-abort-the-merge">Option 1: Abort the merge</a></li>
<li><a href="#option-2-reset-to-the-last-commit">Option 2: Reset to the last commit</a></li>
</ul>
</li>
<li><a href="#delete-a-file-from-git-history">Delete a File From Git History</a>
<ul>
<li><a href="#if-you-dont-push-with---force">If you don't push with --force</a></li>
</ul>
</li>
</ul>
<h1 id="use-vim-for-commits">Use Vim for commits</h1>
<p>Ugh, Nano. Kidding, but seriously here's how to use Vim instead.</p>
<pre><code class="language-bash">git config --global core.editor "vim"
</code></pre>
<hr>
<h1 id="rename-branch">Rename Branch</h1>
<pre><code class="language-bash"># Check out the branch to rename
git checkout development
# Rename the branch
git branch -m master
# Push the change
git push origin :development master
</code></pre>
<p>If you're using BitBucket and it fails to delete the branch, it might be set
as your MAIN branch. You need to change that before it can be deleted. You can
change it in the settings page.</p>
<hr>
<h1 id="roll-back-merge">Roll Back Merge</h1>
<p>If you do a git merge and its really ugly, you might want to undo that.</p>
<p>First you did the commit</p>
<pre><code class="language-bash">git checkout master
git merge --squash development
</code></pre>
<p>Then you got a TON of <code>CONFLICT</code> messages and a really dirty state.</p>
<h2 id="option-1-abort-the-merge">Option 1: Abort the merge</h2>
<p>This one's my preference. There might be a few <code>~</code> files left over to <code>rm</code>
manually.</p>
<pre><code class="language-bash">git merge --abort
</code></pre>
<h2 id="option-2-reset-to-the-last-commit">Option 2: Reset to the last commit</h2>
<p>First, get the commit right before this one from git log</p>
<pre><code class="language-bash">git log -n 1
</code></pre>
<p>Copy the commit hash</p>
<pre><code class="language-bash">git reset --hard &#x3C;commit hash>
</code></pre>
<p>You may need to clean up some files after that.</p>
<p><strong>extra</strong>
Usually, the next thing you wanna do is try the merge again with
<code>--stattheirs</code>, or</p>
<pre><code class="language-bash">git merge --squash --strategy-option=theirs development
</code></pre>
<hr>
<h1 id="delete-a-file-from-git-history">Delete a File From Git History</h1>
<p>It's important to note, as GitHub highlights <a href="https://help.github.com/en/articles/removing-sensitive-data-from-a-repository">in their documentation</a>,
if you've uploaded a secret password or something, you still need to change it.</p>
<p>The file I removed was called <code>sqldata.json</code>, and it was in my project root.
From the project root, I ran:</p>
<pre><code class="language-bash">git filter-branch --force --index-filter \
  "git rm --cached --ignore-unmatch sqldata.json" \
  --prune-empty --tag-name-filter cat -- --all

git push origin --force --all
</code></pre>
<h2 id="if-you-dont-push-with---force">If you don't push with --force</h2>
<p>When I tried to push, though, I got an error. I'd forgotten to use <code>--force</code>.</p>
<pre><code class="language-bash">To https://github.com/kylep/jsc2f.git
 ! [rejected]        master -> master (non-fast-forward)
 error: failed to push some refs to 'https://github.com/kylep/jsc2f.git'
 hint: Updates were rejected because the tip of your current branch is behind
 hint: its remote counterpart. Integrate the remote changes (e.g.
 hint: 'git pull ...') before pushing again.
 hint: See the 'Note about fast-forwards' in 'git push --help' for details.
</code></pre>
<p>Then, when I tried to pull, I got</p>
<pre><code class="language-bash">From https://github.com/kylep/jsc2f
 + ca7d7cc...710d37b master     -> origin/master  (forced update)
 fatal: refusing to merge unrelated histories
</code></pre>
<p>If you're getting errors like that, maybe you need to run push with <code>--force</code>.</p>
<pre><code class="language-bash">git push origin --force --all
</code></pre>


<p><strong>This is a Reference Page:</strong> Reference pages are collections of short bits
of information that are useful to look up but not long or interesting enough to
justify having their own dedicated post.</p>
<p>This reference page contains python-related mini-guides and minor posts.</p>
<hr>
<h3 id="table-of-contents">Table of Contents</h3>
<ul>
<li><a href="#pylint-ignore-comments">Pylint Ignore Comments</a>
<ul>
<li><a href="#turn-off-linting-for-now">Turn off Linting for now</a></li>
</ul>
</li>
<li><a href="#get-an-object-from-list-of-objects-where-the-object-property-has-value-x">Get an object from list of objects where the object property has value x</a>
<ul>
<li><a href="#a-subset-list-is-expected-list-comprehension">A subset list is expected: list comprehension</a></li>
<li><a href="#get-one-match-next-w-generator-expression">Get one match: next() w/ generator expression</a></li>
</ul>
</li>
<li><a href="#iterate-through-dictionary">Iterate through dictionary</a></li>
<li><a href="#print-json-nicely-in-ascii-no-unicode-u-prefix">Print JSON nicely in ASCII (no unicode 'u' prefix)</a></li>
<li><a href="#replace-a-string-in-a-file">Replace a string in a file</a></li>
<li><a href="#get-environment-variable">Get environment variable</a></li>
<li><a href="#use-jinja2-for-templating">Use Jinja2 for templating</a>
<ul>
<li><a href="#jinja2-templating-with-files">Jinja2 templating with files</a></li>
</ul>
</li>
<li><a href="#execute-sql-query-against-mysql-or-mariadb">Execute SQL query against MySQL or MariaDB</a></li>
<li><a href="#read-from-std-in-bash-pipe">Read from std in (bash pipe)</a></li>
<li><a href="#custom-exceptions">Custom Exceptions</a></li>
<li><a href="#cant-run-pip-after-updating">Can't run pip after updating</a></li>
</ul>
<h1 id="pylint-ignore-comments">Pylint Ignore Comments</h1>
<p>I can never remember the syntax to ignore certain Pylint warnings. Yeah you can
turn them off everywhere but usually I just want to ignore a certain occurrence.</p>
<pre><code class="language-python"># - This file is a mess and I don't want pylint here:
# pylint: disable-all

# - troublesome import not being found:
# pylint: disable=import-error

# - redefined-builtin (W0622):
# pylint: disable=W0622

# - too-many-public-methods (R0904):
# pylint: disable=R0903

# - too-many-arguments (R0913):
# pylint: disable-msg=R0913

# - too-many-locals (R0914)
# pylint: disable=R0914

# - Invalid constant name:
# pylint: disable=C0103

# - Catching too general exception:
# pylint: disable=W0703

# - Using python3 compatible print statement makes python2 linter mad
# pylint: disable=superfluous-parens

# - Ignore too-long line (this needs both the pylint command and the trailing no-qa)
# pylint: disable=line-too-long
binaryValue = b'QVdTOmV5SndZWGxzYjJGa0lqb2lVSFkwWXpKeFNEaHZXVkkxVmtwRmVXdFhXamhKUVRGcGNFWk1kelJKZEV0V1UwSk9'  # noqa: E50

# - Wrong Import Position (Required when putting python version check up top)
# flake8: noqa=402
# pylint: disable=wrong-import-position
</code></pre>
<h2 id="turn-off-linting-for-now">Turn off Linting for now</h2>
<p>In VIM, run:</p>
<pre><code>:ALEToggle
</code></pre>
<h1 id="get-an-object-from-list-of-objects-where-the-object-property-has-value-x">Get an object from list of objects where the object property has value x</h1>
<h2 id="a-subset-list-is-expected-list-comprehension">A subset list is expected: list comprehension</h2>
<p>See <a href="https://docs.python.org/3/tutorial/datastructures.html">Section 5.1.3 - List comprehension</a></p>
<pre><code class="language-python"># This will return a list of objects whith a value
[obj for obj in haystack_list if obj.value == search]

# Example of list comprehension to get the even numbers in a list
nums = [1, 2, 3, 4, 5, 6, 7, 8]
even_nums = [num for num in nums if num % 2 == 0]
</code></pre>
<h2 id="get-one-match-next-w-generator-expression">Get one match: next() w/ generator expression</h2>
<p>The <code>next()</code> method will return the first element found in an iterable.</p>
<p>See <a href="https://www.python.org/dev/peps/pep-0289/">PEP 289 - Generator expressions</a></p>
<pre><code class="language-python"># Raises a StopIteration exception if no match is found
needle = next((obj for obj in haystack_list if obj.value == value))

# To return a value instead of raising an exception, pass a second argument
needle = next((obj for obj in haystack_list if obj.value == search), None)

# Example of what Next is really doing. This returns 1.
next(iter([1,2,3]))
</code></pre>
<hr>
<h1 id="iterate-through-dictionary">Iterate through dictionary</h1>
<pre><code class="language-python">ex_dict = {'a': 1, 'b': 2, 'c': 3}
for key in ex_dict:
    print(key)
    print(ext_dict[key]) # value
</code></pre>
<p>This works because <code>ex_dict.keys() == list(ex_dict)</code></p>
<hr>
<h1 id="print-json-nicely-in-ascii-no-unicode-u-prefix">Print JSON nicely in ASCII (no unicode 'u' prefix)</h1>
<p>This is like using <a href="https://docs.python.org/3/library/pprint.html">pprint</a> on
the JSON object but without the unicode prefixes.</p>
<pre><code class="language-python">data = ... # the json string
jdata = json.dumps(json.loads(data),  indent=4, sort_keys=True)
print(jdata)
</code></pre>
<hr>
<h1 id="replace-a-string-in-a-file">Replace a string in a file</h1>
<p>Though honestly, <a href="https://www.geeksforgeeks.org/sed-command-in-linux-unix-with-examples/">sed</a>
is usually a better option. Or if there are a lot of values, I like to use a
jinja template.</p>
<pre><code class="language-python">file_path = '/home/user/example_file'
find_str = 'REPLACE_THIS'
replace_str = 'WITH_THIS'

# Open the file to read the value
with open(file_path, 'r') as read_file:
    file_content = read_file.read()

# Replace the string
file_content = file_content.replace(find_str, replace_str)

# Write the file
with open(file_path, 'w') as write_file:
    write_file.write(replaced_content)
</code></pre>
<hr>
<h1 id="get-environment-variable">Get environment variable</h1>
<p>I mostly use this with <a href="https://docs.docker.com/engine/reference/builder/">Dockerfile ENV values</a>.</p>
<pre><code class="language-python">import os

environment_var_name = 'MY_ENV_VAR'
environment_var_vlue os.environ[environment_var_name]
</code></pre>
<hr>
<h1 id="use-jinja2-for-templating">Use Jinja2 for templating</h1>
<p>I use this for config files, like <a href="https://docs.ansible.com/ansible/latest/user_guide/playbooks_templating.html">how Ansible does it</a>.</p>
<pre><code class="language-python">from jinja2 import Template

replacements = {'USERNAME': 'kyle', 'PASSWORD': 'example'}
template_text = 'connection = mysql://{{USERNAME}}:{{PASSWORD}}@127.0.0.1'
template = Template(template_text)
replaced_text = template.render(**replacements)
</code></pre>
<h2 id="jinja2-templating-with-files">Jinja2 templating with files</h2>
<p>Same idea as above but it reads a .j2 file as input and writes it to a config
file. Can combine this nicely with environment variables to write config files
as part of a Docker start script.</p>
<pre><code class="language-python">from jinja2 import Template

def apply_template(jinja2_file, output_file, replacements):
    """Replace the replacements values in jinja2_file, write to output_file"""
    with open(jinja2_file, 'r') as j2_file:
        j2_text = j2_file.read()
    template = Template(j2_text)
    replaced_text = template.render(**replacements)
    with open(output_file, 'w+') as write_file:
        write_file.write(replaced_text)

</code></pre>
<hr>
<h1 id="execute-sql-query-against-mysql-or-mariadb">Execute SQL query against MySQL or MariaDB</h1>
<pre><code class="language-python">import mysql.connector as mariadb

conn = mariadb.connect(host=db_host, port=db_port, user=db_username,
                           password=db_password, database=db_name)
cursor = conn.cursor()
cursor.execute(sql_query)
conn.commit() # if it was a write action
conn.close()
</code></pre>
<hr>
<h1 id="read-from-std-in-bash-pipe">Read from std in (bash pipe)</h1>
<p>For instance I have a file named x that has a big string where I want to
replace commas with spaces. Sed works but I can never remember the syntax.</p>
<pre><code class="language-python">cat x | python -c "import sys; print(sys.stdin.read())"

# Example of using python instead of sed to replace , with spaces
cat x | python -c "import sys; print(sys.stdin.read().replace(',',' '))"

# You can use this in fun ways like combining it with bash loops
cat x | python -c \
  "import sys; print(sys.stdin.read().replace(',','\n'))" | while read line; do
  echo $line
done
</code></pre>
<hr>
<h1 id="custom-exceptions">Custom Exceptions</h1>
<pre><code class="language-python">class MyCustomException(Error):
    pass
</code></pre>
<hr>
<h1 id="cant-run-pip-after-updating">Can't run pip after updating</h1>
<p>So you ran into a problem because ubuntu ships an ancient version of pip, and
some dependency package like <code>subprocess32</code> is breaking your install. A forum
post says you should update pip.</p>
<pre><code class="language-bash">pip install -U pip
</code></pre>
<p>Except now pip won't run at all.</p>
<pre><code class="language-bash">pip install python-openstackclient
Traceback (most recent call last):
  File "/usr/bin/pip", line 9, in &#x3C;module>
    from pip import main
ImportError: cannot import name main
</code></pre>
<p>I think maybe the pip executable file moved after the update.
To fix it:</p>
<pre><code class="language-bash">hash -r pip
</code></pre>
<p>Now pip works again.</p>
<p>As an aside, if <code>subprocess32</code> really was your problem, updating pip didn't
help. For some reason you have to use apt to install <code>python-subprocess32</code>.</p>


Kyle Pericak

Ceph Reference Page

Error Messages, Codes, & Stack Traces

Operations Reference Page

Git Reference Page

Python Reference Page